Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GPT3 MoE #4

Open
wants to merge 53 commits into
base: main
Choose a base branch
from
Open

GPT3 MoE #4

wants to merge 53 commits into from

Conversation

haeggee
Copy link

@haeggee haeggee commented Aug 5, 2024 •
edited
Loading

WIP

  • new model class with moe changes on top of gpt3
  • aux losses with backward through pipeline engine
  • correct compute_block_costs and flops
  • moe with mlp that is not 3 weight matrices (i.e. not llama style)
  • sparse upcycling / load from pretrained
  • hf conversion

AleHD and others added 30 commits June 26, 2024 12:38
@AleHD
XGLM work in progress: Causal Attention and Positional Embeddings work
7aa3940
@AleHD
WIP: GPT arch almost done, hf->nt converters working perfectly for no…
78dd53c 
…n-distributed inference
@AleHD
Added hf2nt frontend + tested training
a74c71a
@AleHD
Added nt2hf conversion + tests :)
04eaef9
@AleHD
precommit
138da5f
@negar-foroutan
Merge pull request #1 from swiss-ai/gpt
35c43f7 
GPT-3 and XGLM implementation
@TJ-Solergibert
Added MultilingualNanoset Config
0485fd6
@TJ-Solergibert
Added MultilingualNanoset
539832a
@TJ-Solergibert
Added Language token
d9f0670
@TJ-Solergibert
Forgot the trainer ups
efe8720
@TJ-Solergibert
Fix minor errors. Everything works
25ad39b
@TJ-Solergibert
Updated config file with GPT2 tokenized datasets in RCP
d91f9e1
@TJ-Solergibert
Before lunch
d0c14e3
@TJ-Solergibert
After lunch
9cfc5ea
@TJ-Solergibert
Ready
eed7bce
@negar-foroutan
Merge pull request #2 from TJ-Solergibert/multilingual_nanoset
da50231 
Multilingual Nanoset
start documenting moe setup
7a932f8
@haeggee
base moe file
f08a05e
@haeggee
add todo
fa06c0d
@haeggee
gpt3_moe basis
a9dba53
@haeggee
add nn.linear to init for moe router
2efffb8
@haeggee
changes to pipeline for backward through aux losses
57c58b7
@haeggee
correct block costs and flops
3967bee
@haeggee
case of dict in pipelineblock
bcb94cc
@haeggee
option for GLU or normal MLP
91acdc0
@haeggee
init of linear layer in starcoder
df3befc
@haeggee
potential bug in pipeline block
6edce83
@AleHD @negar-foroutan
XGLM work in progress: Causal Attention and Positional Embeddings work
7425167
@AleHD @negar-foroutan
WIP: GPT arch almost done, hf->nt converters working perfectly for no…
42695ba 
…n-distributed inference
@AleHD @negar-foroutan
Added hf2nt frontend + tested training
6294aad
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@haeggee @AleHD @negar-foroutan @TJ-Solergibert